Questions and instruments

Question writing

Problematic Questions

How could we improve this question?

Question: Do you support or oppose restoring the Joint Comprehensive Plan of Action (JCPOA) with Iran? (Answers range from strongly support to strongly oppose)

Problematic Questions: lack of context

  • Even people who are familiar with the deal may not know the formal name, the “Iran nuclear deal” might be more common.

  • If feasible, a description of the policy may be more useful than just the name.

Yougov Iran nuclear deal question

Problematic Questions

Are we spending too much money on welfare and national defense, or not enough? (Answers range from “too little” to “too much”)

Problematic Questions: double-barreled

  • I might want to spend more on welfare and less on national defense, or vice versa.
  • At a minimum, we probably want to split these up. (The General Social Survey asks a whole series of spending questions like this)

GSS Spending Questions

Problematic Questions

Have you ever joined a political protest? (options are “I have done this”, “I might do this”, or “I would never do this”)

Problematic Questions: non-exclusive categories

  • These response options aren’t clearly mutually exclusive: I may have participated in the past but be unwilling to do so in the future. Or vice-versa.
  • Splitting this up or adding a full range of options could be better.

Question from the 1992 ISSP

Problematic Questions

How often do you drink alcohol? (options range from “Never” to “Frequently”)

Problematic Questions: ambiguous time frames

  • Respondents can interpret the relevant time frame differently. Lifetime? In the last week?
  • Specifying some time frame can give you more consistent answers.

Question from NHANES

Problematic Questions

In the last six months, have you been a victim of a crime? (options are “yes” or “no”)

Problematic Questions: context and recall

  • Respondents may not see themselves as victims, may forget minor crimes, or may not immediately think of even major crimes as “criminal”.
  • Surveys like the NCVS use examples and ask about specific aspects of each crime.

NCVS alternative theft questions

Problematic Questions

Do you abuse illegal drugs? (responses are “yes” or “no”)

Problematic Questions: loaded language

  • Explicitly referencing loaded terms can make people less willing to give honest answers, especially on sensitive questions.
  • Respondents may genuinely not consider themselves drug abusers.

Marijuana questions on the Youth Risk Behavior Survey

Problematic Questions

Do you blame Democrats for the government shutdown? (Answers are “Yes”, “No”, “Not sure”)

Problematic Questions: leading

  • Mentioning one party here is likely to nudge respondents toward blaming Democrats
  • The response options would fail to capture the possibility of assigning equal blame to both parties.

Yougov’s shutdown question

Question writing

There are no hard and fast rules here, sometimes a flawed question is still the best possible option, but there are some general best practices

Question writing

  • Avoid ambiguous or emotionally charged language, especially when asking about a sensitive topic.
  • Provide context where needed, and choose familiar names instead of jargon.
  • Be specific about time frames.
  • Offer response options that are exhaustive and mutually exclusive.
  • Ask questions that only measure one “thing” at a time.
  • Keep things short.

Question writing

Where possible, you should try to use an established measure instead of reinventing the wheel.

How respondents think about questions

Social Desirability

Everyone wants to be liked. Respondents will give answers that make them look good. Maybe even subconsciously.

Social Desirability: voting

  • As many as half of non-voters will claim to have voted in survey questions.

  • (Hanmer, Banks, and White 2014) randomly gave subtle and unsubtle hints that researchers might check voter roles.

Social Desirability: voting

  • Perhaps unsurprisingly, the desire to avoid being caught in a lie overwhelmed social desirability.

(Hanmer, Banks, and White 2014)

Social Desirability: list-experiments

List-experiments are another option for dealing with sensitive topics by indirectly measuring preferences.

Social Desirability: list-experiments

  • Respondents are randomly assigned to receive only non-sensitive items, or non-sensitive items plus one sensitive question.

  • They only say how many items they agree with.

(Kuklinski et al. 1997)

Social Desirability: list-experiments

  • The difference in the average count of items is used to estimate support.

(Kuklinski et al. 1997)

Social Desirability: some fixes

  • Taking the interviewer out seems to help a lot! This is less of a problem on online surveys.
  • Deliberately loading questions is another option (see the example marijuana question from the YRBSS)
  • For highly sensitive topics, list experiments can mitigate the problem.

Fence siting

  • Some respondents may always pick the “neutral” option, if one is available.

  • One option is to eliminate the neutral option all together, but there’s a trade-off here: some people genuinely don’t have an opinion!

Acquiescence/naysaying

  • Some respondents may be prone to agree or disagree with everything. This could be because of their personality or a lack of attention.

Acquiescence/naysaying: F-scale

  • The F-scale was developed out of an early effort to understand the psychology of rank and file fascists.
  • 30 agree/disagree items, with agreement indicating more authoritarian views.
  • Acquiescence bias would probably inflate the estimates of authoritarian attitudes.

Acquiescence/naysaying: some fixes

  • Balanced agree/disagree items: agreement could = more of a characteristic in some questions, and less of it in others. (“you can’t trust strangers” and “most people are honest”)

  • Forced choice: instead of agree/disagree, use either/or (“is it better to be obedient or creative?”) or some other item-specific options (“always - never”, “many - none”, etc.)

Careless responding

  • Respondents may get worn out, bored, or not really care to give accurate answers.

  • (this can be especially problematic for surveys that offer rewards for respondents.)

Careless responding

  • In a 2022 online opt-in survey, Pew found that 5% of adults said they were licensed to operate a nuclear submarine.
  • The actual number would round to 0%

Pew

Careless responding: some fixes

  • Limiting the number of questions can help mitigate boredom.
  • Including some attention screens (questions with clear answers that disqualify inattentive respondents) or scales with some reverse-coded items.

Multi-item measures

  • Complex concepts often require more than one thing to measure appropriately.

  • This can reduce measurement error, but it also gives us some options for validating measurements.

Racial resentment

  • Reported “old-fashioned” racism declined precipitously after the Civil Rights movement.

  • But most Americans continued to oppose programs designed to improve racial equality.

  • Symbolic racism (or racial resentment, or modern racism) is a proposed explanation for this phenomenon.

Symbolic Racism

Described as a blend of “anti-black affect” and “traditional conservatism” with four components:

  • “racial discrimination is no longer a serious obstacle to blacks’ prospects for a good life”
  • “blacks’ continuing disadvantages are largely due to their unwillingness to work hard enough.”
  • “As a result, both their continuing demands and increased advantages are unwarranted”

Scale: Symbolic Racism 2000 scale

(Henry and Sears 2002)

Symbolic Racism: Critiques

Some major critiques:

  • Questions are tautological

  • The questions mostly just measure conservatism.

  • The construct isn’t really distinct from old-fashioned racism.

References

Hanmer, Michael J, Antoine J Banks, and Ismail K White. 2014. “Experiments to Reduce the over-Reporting of Voting: A Pipeline to the Truth.” Political Analysis 22 (1): 130–41.
Henry, Patrick J, and David O Sears. 2002. “The Symbolic Racism 2000 Scale.” Political Psychology 23 (2): 253–83.
Kuklinski, James H, Paul M Sniderman, Kathleen Knight, Thomas Piazza, Philip E Tetlock, Gordon R Lawrence, and Barbara Mellers. 1997. “Racial Prejudice and Attitudes Toward Affirmative Action.” American Journal of Political Science, 402–19.